The Behavior Analyst Today 


Volume 9, Issue 1 , 2008 


The Functional Analytic Psychotherapy Rating Scale (FAPRS): 
A Behavioral Psychotherapy Coding System 

Glenn M. Callaghan William C. Follette, L. E. Ruckstuhl, Jr., & Peter J. N. Linnerooth 

Abstract 

Many researchers and clinicians believe that the therapeutic relationship is essential in bringing 
about clinical change. Empirical research to support this contention is scarce in part due to the 
difficulty of specifying and measuring theoretically derived mechanisms of change and the important 
dimensions of the client-therapist relationship. Functional Analytic Psychotherapy (FAP; 

Kohlenberg & Tsai, 1991) is a behavioral treatment that delineates how the therapeutic relationship 
brings about clinical change in clear and measurable terms. While initial research has been conducted 
to demonstrating the effectiveness of FAP with different populations, the purported mechanism of 
clinical change in FAP has not been sufficiently documented. This study describes the creation of a 
behavioral coding system (the Functional Analytic Psychotherapy Rating Scale; FAPRS) to identify 
and specify the components believed to be essential in bringing about client behavior change in 
FAP. Interobserver agreement values indicated moderate to high levels of reliability for the coding 
system. Implications for future tests of FAP’s proposed mechanism of change and the validity of the 
coding system are discussed. 
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Psychotherapy is a set of circumstances that can produce meaningful, lasting and positive 
behavior change. It is unclear, however, which aspects of many treatments actually cause change. Most 
therapies assume that the manipulation of certain critical variables in psychotherapy, called mechanisms of 
change, will reduce client problem behaviors and increase adaptive behaviors. Although mechanisms of 
change are assumed to operate in the day-to-day practice of psychotherapy, they are rarely studied 
empirically, and are often left either unspecified or defined in a way that makes detection and 
measurement of their operation difficult. The client-therapist relationship is one such potential mechanism 
of change in need of further specification and study. 

Psychotherapy has been described as the “systematic use of a human relationship for therapeutic 
purposes” (Butler and Strupp, 1986, p. 36). Many other therapists and researchers also regard the 
therapeutic relationship as essential to the process of client change (Beck, Rush, Shaw, & Emery, 1979; 
Fiedler, 1950; Greenberg, 1994; Hentschel & Bijleveld, 1995; Horvath & Luborsky, 1993; Krasner, 1962, 
1963; Lazarus, 1972; Morganstem, 1988; Rimm & Masters, 1974; Rogers, 1957; Rosenfarb, 1992; 

Schaap, Bennun, Schindler, & Hoodguin, 1993; Wright & Davis, 1994). There is an important gap, 
however, between assumptions about the importance of the therapeutic relationship and empirical 
documentation of the events that change client behavior. 

To empirically establish a link between the therapeutic relationship and client behavior change, 
psychotherapy researchers must accomplish several tasks. First, a treatment must be identified and 
described. Second, the hypothesized mechanism of change for that treatment must be specified in a way 
that is clear and measurable. Third, the dimensions of behavior that are hypothesized to change following 
the manipulation of the mechanism must be defined. Fourth, a measurement system that allows reliable 
observation of behavior occurring within that treatment must be designed. Fifth, the validity of the 
hypothesized mechanism of change must be tested by empirically linking the manipulation of that 
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mechanism to client improvement. This improvement should be documented within the therapeutic 
relationship, and ultimately it should be shown to generalize to other relationships the client has outside of 
therapy. 


This article presents a method for accomplishing the goals described above for Functional Analytic 
Psychotherapy (FAP; Kohlenberg & Tsai, 1991) an interpersonally oriented, behavioral psychotherapy. It 
defines FAP’s mechanism of change (therapist contingent responding; discussed below) and the 
dimensions of client behavior that are impacted by the therapist’s contingent responses in -session. It 
describes the development of and presents data on a reliable coding system for FAP (the Functional 
Analytic Psychotherapy Rating Scale; FAPRS; Callaghan, 1998). Finally, it presents initial data testing the 
validity of FAP’s hypothesized mechanism of change. 

The primary purpose of this study was to create and evaluate a coding system that could reliably 
demonstrate FAP’s mechanism of change (and the client behaviors hypothesized to be impacted by it). In 
the next section, the important dimensions of the therapeutic relationship in FAP will be introduced and 
FAP’s mechanism of change, therapist contingent responding, will be defined. 

Functional Analytic Psychotherapy 

Functional Analytic Psychotherapy (FAP; Kohlenberg & Tsai, 1991; see also Kohlenberg, Hayes 
& Tsai, 1993) is used primarily to treat psychological disorders that are interpersonally based, such as 
those with pervasive repertoire deficits including generalized anxiety disorder, dysthymia, social phobia, 
and the Axis-II personality disorders (American Psychiatric Association, 1994). FAP assumes that the 
interpersonal problems that clients have in relationships outside of therapy will also occur in the context of 
the therapeutic relationship. The FAP therapist focuses on creating clinical change in this context. The 
therapist can influence client problem behaviors and improvements because he or she has direct access to 
the contingencies that occur in-session. Not all behaviors occur in the therapeutic relationship exactly as 
they do outside of session. However, the FAP therapist attempts to understand these behaviors 
principally, or functionally, and looks for opportunities to respond to them in-session, rather than dealing 
exclusively with events that happened outside of therapy. 

Although researchers have only just begun to demonstrate the effectiveness of FAP in treating 
different problems (Gaynor, 2002; Kohlenberg, et al., 2004), treatments for depression (Kohlenberg, 

Kanter, Bolling, Parker, & Tsai, 2002), adolescent behavioral problems (Gaynor & Lawrence, 2002), 
personality disorders (Callaghan, Summers, & Weidman, 2003), and the use of FAP as an adjunctive 
treatment for post traumatic stress disorder (Prins & Callaghan, 2003) have already been undertaken. 

Data from these studies indicate mounting evidence for the effectiveness of this treatment with diverse 
populations and a strong need to attempt to document the mechanism of change in FAP. 

Mechanism of Change 

The conceptual literature on FAP has identified therapist contingent responding, a specifiable and 
measurable aspect of the client- therapist relationship, as FAP’s mechanism of client change. Therapist 
contingent responses occur contingent on clients’ in-session, clinically relevant behaviors (Callaghan, 1996; 
Callaghan, Naugle, & Follette, 1996; Follette, Naugle, & Callaghan, 1996; Kohlenberg & Tsai, 1995). 
Clinically relevant behaviors are clients’ specified problem behaviors, targets for improvement, and 
descriptions of functional relationships between their own behavior and its antecedents and consequences 
and are described in detail below. 
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Clinically Relevant Behaviors 

In order to understand what maintains the problems that brought the client to therapy and to plan 
an effective treatment, FAP therapists conceptualize clients’ interpersonal behavioral repertoires in the 
context of the developing therapeutic relationship. FAP therapists look for and respond to three types of 
clinically relevant behaviors (CRBs) that occur in therapy. In general, clinically relevant behaviors can be 
described as behaviors that occur within the context of the therapeutic relationship that have an identifiable 
(desirable or undesirable) function on the therapist. 

Clinically Relevant Behavior 1. Client problem behaviors, CRB Is, are the responses that clients 
emit in-session that contribute to the client’s problems in living outside of treatment. CRB Is are often 
behavioral excesses that aversively stimulate persons in the client’s life, but may also be behavioral or 
motivational deficits that negatively impact the client’s social relationships. For example, a FAP client may 
present for therapy complaining that he or she would like to develop closer, more meaningful interpersonal 
relationships, but fears doing so because he or she may be rejected and feel hurt. A CRB1 for such a 
client would be any escape or avoidance behavior in the therapeutic relationship that prevents the client 
from developing a better interpersonal relationship with the therapist. 

Clinically Relevant Behavior 2. Client improvements, CRB2s, are behaviors that occur in the 
context of the therapeutic relationship that are effective for the client given his or her goals for therapy. 
Initially, many FAP clients have repertoire or motivational deficits in areas important to good interpersonal 
relationships. The FAP therapist’s job is to help the client construct an effective repertoire starting with 
the client’s existing skills (e.g., Goldiamond, 1974). Continuing the above example, any client behavior that 
serves to create a meaningful, close relationship with the therapist, such as appropriate disclosure or 
displaying affect that feels “risky” to the client, would be identified as a CRB2 for the client. 

Clinically Relevant Behavior 3. Client verbal descriptions of controlling variables, or CRB3s, are 
statements of the functional relationship between the client’s behavior and the social-environmental 
antecedents and consequences of that response. Continuing with the above example, the client might say 
to the therapist, “The way that you talk to me shows that you care, even when you don’t like what I’ve 
just done or said. This tells me I can take risks, or even fail miserably. You won’t stop caring about me, 
even though I sometimes do things that are insensitive to you. That’s why I like talking to you.” Here, the 
client has emitted a CRB 3 because he or she has described the function of an antecedent, the therapist’s 
history of responding in difficult discussions, on his or her own current, private evaluation of the 
therapeutic relationship. The client has specified the conditions that make it more likely he or she will 
engage in a particular response with the therapist. 

Functional case conceptualization. CRBs are defined according to their function on the 
therapist, and are formally organized in a functional case conceptualization based on behavioral principles. 
The case conceptualization describes the current interpersonal repertoire of the client as it is emitted 
within the therapeutic relationship. The conceptualization is used to guide an idiographic assessment of 
client behavior and corresponding intervention strategies. Functional case conceptualizations are iterative. 
They are updated as new client target behaviors are identified and as behavioral improvements are 
established. 

The FAP coding system to be described below was designed to recognize the function of the 
behavior of both members of the therapeutic dyad. It was also designed to incorporate important 
contextual variables, and to draw functional distinctions among therapists’ responses to client CRBs. In the 
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following section we will expand on our definition of therapist contingent responding and reveal some of 
the therapeutic process of FAP by presenting how FAP therapists decrease problem behaviors and shape 
or maintain improvements. 

In-Session Contingent Responding in FAP 

Therapist contingent responding is the contingent application of the therapist’s natural (Ferster, 
1967, 1972) reaction to the client’s behavior as it occurs in session. Contingent responses are delivered 
based on how the client’s behavior actually impacts the therapist (negatively or positively), not simply on 
the appearance of the behavior. When a FAP therapist detects the occurrence of CRBs in session, he or 
she seeks to make an immediate, contingent, and appropriate therapeutic response in order to impact the 
behavior as it occurs. This may include responses that appropriately decrease the frequency or intensity 
of, or extinguish altogether, problem behaviors. These responses are then combined with therapist 
behaviors that evoke and reinforce client improvements. The FAP therapist’s goal is to affect the 
frequency and shape the function of the client’s in-session behavior in ways that will improve both the 
therapeutic relationship and generalize to serve the client’s goals in outside relationships. 

Decreasing problem behaviors. To decrease problematic in session behavior (CRB1), FAP 
therapists can ignore the maladaptive response, in which case the absence of reinforcement leads to 
extinction of the problem behavior, or the response can be contingently punished. Examples of contingent 
punishers include explicitly stating the negative impact that the client’s behavior has on the therapist in that 
moment during session. In the coding system to be described, such a response (an explicit therapist 
behavior that functions to decrease the future probability of CRB1) is called a TCRB1. 

Shaping improvements. Extinction or punishment of maladaptive behaviors is never used 
alone. Following their contingent response to a CRB 1 , FAP therapists immediately attempt to constmct 
adaptive, alternative behaviors (CRB2s) that will be more useful for the client (e.g., Hawkins, 1986). 
Immediately evoking and reinforcing a client CRB2 are particularly salient in the moments following the 
therapist’s response to a CRB1. Thus, FAP therapists constantly seek to detect or evoke and then 
differentially reinforce adaptive alternative client behaviors. Consistent, differential reinforcement of 
alternatives to CRB Is allows the FAP therapist to shape functional improvements (CRB2s) in the client’s 
repertoire, rather than just decrease the frequency of symptoms or maladaptive social behaviors. 

Drawing functional connections. In the moments following a CRB1, the therapist may also 
choose to model a description of controlling variables (CRB3) for the client’s behavior by pointing out the 
relationship between maladaptive behaviors in session, their function on the therapist, and their relation to 
aversive consequences that the client has experienced in other relationships. Client behaviors that evoke 
therapist escape, avoidance or countercontrol are likely to have the same function on significant others in 
the client’s natural social environment. Although FAP therapists take as their first priority the direct 
training of improvements (evoking and reinforcing CRB2), drawing functional connections between client 
behavior in- and out of session is an important element of FAP. 

FAP does not attend solely to client problem behaviors, psychological symptoms and distress in 
therapy sessions. FAP’s focus on the development of a prosocial client repertoire marks it as a 
constructionist approach to therapy. FAP builds on existing client repertoires to enhance functioning, and 
FAP therapists explicitly conceptualize client strengths throughout treatment. Evoking and differentially 
reinforcing approximations toward more effective responding enhances the client’s interpersonal repertoire 
and the therapeutic relationship. This two-pronged approach to therapist contingent responding is FAP’s 
mechanism of client behavior change. 
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Initial Research on the Mechanism of Change in FAP 

The purpose of this study was to begin the empirical documentation of FAP’s mechanism of 
change. The study was thus designed to accomplish two goals: (1) to create a coding system capable of 
documenting both client and therapist behavior in FAP sessions and; (2) to demonstrate that observers 
using the coding system can reliably track the occurrence of client and therapist behavior in session. Most 
importantly this study sought to reliably document the occurrence of CRBs and therapist contingent 
responses to CRBs. Because reliability is a necessary precursor to validity in any measurement system, 
the data presented in this study represent a crucial first step toward conducting tests of the model of 
change proposed for FAP and the implementation of efficacy and effectiveness trials. Because there was 
no coding manual for FAP prior to this research, the development of the Functional Analytic 
Psychotherapy Rating Scale (FAPRS) will be described. Data demonstrating the reliability of the FAPRS 
system, using measures of intra- and interobserver agreement, will then be presented. 

Functional Coding with the FAPRS 

The FAPRS codes the function of client or therapist behavior rather than the form. Raters using 
the FAPRS must recognize the function of in session behavior by its effect on the other member of the 
client-therapist dyad, rather than simply noting the occurrence of a particular topography. As a simple 
example, two clients might ask their therapist “What did you do over the weekend?” Even though the 
same question is asked, it could not be coded topographically because an identical question would have a 
different function depending upon the conceptualization of the client asking it. For a client who has had 
difficulty showing caring within interpersonal relationships, asking such a question would be an 
improvement (CRB2). In another instance, however, the question may be representative of a client 
problem (CRB1) that inhibits the development of the therapeutic relationship and client progress by 
allowing the client to forestall, avoid, or escape discussing an important event that occurred during the time 
between sessions with the therapist. When using the FAPRS a client’s responses is coded based on the 
function the behavior has on the therapist, given the client’s specific case conceptualization. 

As with the client’s behavior, the therapist’s responses are coded based on the function or impact 
the behavior has on the client. The therapist’s responses are not coded based on how the behavior 
appears (e.g., making an “I-statement”), nor are they coded based on what the rater believes the 
therapist’s intentions were when making the response. Effectiveness is determined by the function of each 
therapist response on the client given the case conceptualization. Therapist responses that reinforce or 
shape the quality of a client’s ability to show caring or willingness to share reports of important in-session 
events would be coded as effective responses by the FAPRS. Missing a client’s approximation to positive 
change or allowing a client to avoid sharing important information would both be coded as ineffective 
therapist responses. 


RESEARCH DESIGN AND METHODS 
The FAPRS Coding Manual 

Development of Observational Codes and the FAPRS Manual 

Reliable assessment of both members of a therapeutic dyad’ s behavior is a necessary foundation 
for future studies designed to determine the empirical relationship between specific types of therapist 
responding and client behavior change. It is also necessary to document FAP’s mechanism of change. 

The FAPRS thus includes both client and therapist behavior codes. The FAPRS was created by extending 
a set of codes designed to identify therapist behavior employed in a previous empirical study (Callaghan & 
Follette, 1999). 
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Instructions for raters. The Functional Analytic Psychotherapy Rating Scale (FAPRS) employs 
the speaking “turn” (also called a “floor change”) as its unit of analysis. Raters are instructed in 
procedures for turn-by-tum coding of sessions using session transcripts and videotapes. Raters are 
allowed to apply only one code to each turn and are provided with a multiple code decision hierarchy for 
use in situations where more than one codeable event occurs within a turn. The FAPRS gives raters 
examples, counter examples, and marginal examples for each code. Raters are instructed to code every 
turn based on the approximate function of the behavior in that turn given the case conceptualization for 
each client. 

Client codes. The client codes used for this study are presented in Table 1. This set of codes 
indicates whether the client is engaging in the FAP-specific, clinically relevant behaviors (CRB1, CRB2, or 
CRB3) versus behaviors likely to occur in other therapies (CTR, 01, 02, CPR). For example, if the client 
talks about the therapeutic relationship, but this behavior is not a specific problem or improvement (CRB), 
it is coded using the Client Focus on the Therapeutic Relationship (CTR) code. When clients emit 
responses that are not improvements or problem behaviors (in or outside of session) but are part of general 
therapy 


Table 1. List of Client Codes in FAPRS 


Code 

Full Name of Code 

Brief Description 

CRB1 

Clinically Relevant Behavior 
1 (problems in session) 

client engages in problematic behavior in-session in the 
context of the therapeutic relationship 

CRB 2 

Clinically Relevant Behavior 
2 (improvements in session) 

client engages in improved behavior in-session in the 
context of the therapeutic relationship 

CRB 3 

Clinically Relevant Behavior 
3 (description of important 
controlling variables) 

client describes how different controlling variables impact 
the client’s behavior and when the client makes these 
functional descriptions 

CTR 

Client Focus on the 
Therapeutic Relationship 

client focuses on the therapeutic relationship 

01 

Discussion of Clinical 
Problems Outside the 
Therapeutic Relationship 
(“outside CRB Is”) 

client discusses or describes problem behaviors that have 
been the focus of treatment but that occur in other 
situations outside of session 

02 

Discussion of Clinical 
Improvements Outside the 
Therapeutic Relationship 
(“outside CRB2s”) 

client discusses or describes improvements that have 
been the focus of treatment but that occur in other 
situations outside of session 

CPR 

Client Positive Session 
Progression 

client discusses or describes problems as they occur in 
situations other than the therapeutic relationship, or 
clarifies or provides context about problems; client 
engages in generally on-task behavior that is facilitative to 
discussion 


progression, the turn is coded as “Client Session Progression” (CPR). When the client tells the therapist 
about an occurrence of a clinically relevant behavior outside of the therapy session (i.e. behavior that is a 
target for therapeutic change), it is coded as an “Outside CRB1” (01) or “Outside CRB2” (02). Ols and 
02s are not CRBs because they do not occur within the therapeutic relationship. 
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FAP assumes that problems and improvements occurring between the client and therapist are 
directly relevant to client problems and improvements in relationships outside of therapy. Ols and 02s are 
important to document because their occurrence indicates the extent to which the client’s behavior 
targeted for change in session generalizes outside the therapeutic relationship. 

Therapist codes. The therapist codes are presented in Table 2. The FAP-specific therapist 
behavior codes (TCRB1, -2 and -3, ECRB, Ml, -2, and -3) are designed to document therapists’ 
contingent responses to clients’ in-session, clinically relevant behaviors. TCRB1, TCRB2 and TCRB3 
indicate effective therapist responses to client in -session CRBs 1, 2 and 3 respectively. Ineffective 
therapist behaviors are noted with the codes Ml, M2 and M3, indicating that the therapist has “missed” a 
CRB occurring in-session. Misses may be due to an inability to discriminate or failure to respond to a 
CRB. Therapist behavior that brings a 


Table 2. List of Therapist Codes in FAPRS 


Code 

Full Name of Code 

Brief Description 

TTR 

Therapist Focus on 
Therapeutic Relationship 

therapist continues focus on therapeutic relationship 
including sharing the therapist’s feelings in response to 
the client 

ECRB 

Therapist Evokes a CRB 

therapist evokes a clinically relevant behavior by the 
client, either CRB1, 2, or 3 

TCRB1 

Therapist Responds 
Effectively to CRB 1 

therapist’s response is to in-session client problem 
behavior 

TCRB2 

Therapist Responds 
Effectively to CRB2 

therapist responds effectively to in-session improvements 

TCRB3 

Therapist Responds 
Effectively to CRB3 

therapist responds to the client describing how different 
controlling variables impact the client’s behavior; 
therapist shapes or models CRB3 for client 

ROl 

Therapist Responds to 
Client’s Discussion of 
Clinical Problems Outside 
the Therapeutic Relationship 

therapist comments on problem behaviors the client 
describes having engaged in outside of the therapy 
session that are a focus of therapy 

R02 

Therapist Responds to 
Client’s Discussion of 
Clinical Improvements 
Outside the Therapeutic 
Relationship 

therapist provides verbal reinforcement in response to 
the client describing improved behaviors outside of the 
therapy session that are a focus of therapy 

TPR 

Therapist Positive Session 
Progression 

therapist engages in generally effective or facilitative 
behavior 

Ml 

Therapist Misses/Does not 
Respond to CRB 1 

therapist does not respond to or misses an opportunity to 
respond to a CRB 1 

M2 

Therapist Misses/Does not 
Respond to CRB 2 

therapist fails to reinforce an instance of a client’s CRB 2 
or a reasonable approximation of a CRB2 

M3 

Therapist Misses/Does not 
Respond to CRB 3 

therapist misses an opportunity to respond to a client’s 
description of important controlling variables or 
reasonable approximation 

IN 

Generally Ineffective 

therapist engages in generally ineffective behavior 
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Therapist Responding 


CRB into the session is noted with the code, Evokes CRB (ECRB). The codes ROl and R02 denote 
effective responses to client descriptions of problem behaviors or improvements occurring outside of the 
therapeutic relationship. TPR indicates general therapist progression. TTR denotes clinician focus on the 
therapeutic relationship that is not in response to a targeted CRB, such as describing the process of FAP 
to the client. Ineffective contingent responding (ICR) and generally ineffective therapist behavior (IN) 
codes capture therapist behavior that is not beneficial (for example, inappropriately arbitrary reinforcers or 
punishers) to the client given the goals of treatment using FAP. 

Reliability Statistic 

Interobserver agreement was determined for all reliability analyses using Cohen’s (1960) kappa. 
The kappa statistic is a measure of inter-rater reliability that improves on percent agreement by correcting 
for agreement that occurs by chance due to high base rates of some codes compared to others. Fleiss’s 
(1981) guidelines for interpreting kappa were used to judge obtained coefficients as “poor” (less than .40), 
“fair” (.40 to .60), “good” (.60 to .75), or “excellent” (above .75) for all analyses presented below. A 
kappa value of .60 or higher (at least “good”) was chosen to define the minimum acceptable level of rater 
competence using the FAPRS. 


Participants 

In the following discussion participants will be named and numbered to differentiate the Clients (1, 
2, 3) Therapists (1, 2) and Raters (1-6). Client-therapist pairs will be referred to as “Dyads” and labeled 
alphanumerically (A, B, C). The term “Cohort” will refer to the groups (1, 2, 3) of raters trained together. 
Cohorts are distinguished because not all raters were trained at the same time. 

Clients 

The 3 clients in this study were adult women with long standing problems across multiple 
interpersonal relationships. The clients sought psychotherapy services in university clinics and provided 
informed consent for psychological treatment and for the use of videotapes and other session materials for 
research. 

Therapists 

Therapist 1 was one of the co-originators of FAP and was considered an expert in conducting the 
treatment. Therapist 1 conducted FAP with 2 of the 3 clients (Client 1 and Client 2). Therapist 2 was a 
doctoral student in clinical psychology with 3 years of experience in FAP and conducted treatment with 
the third client (Client 3). 

Raters 

Rater selection. All raters (1-6) are numbered in descending order according to their experience 
level both in conducting FAP and in using the coding manual. Only raters fa mi liar with the basic principles 
of FAP were selected to code tapes for the study. This decision was based on the assumption that the 
functional nature of the coding system requires raters to understand the behavior analytic distinction 
between how client or therapist responses function as opposed to the superficial appearance, or 
topography, they may have. 
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Criterion rater. Rater 1 was an expert FAP therapist and developed the FAPRS. Rater 1 was 
thus designated the “criterion” rater for this study. The performance of Raters 2-6 was compared to Rater 
l’s in some of the reliability analyses to assess their expertise in the use of the FAPRS. 

Rater training. All raters received 70-100 hours of training. Training began with individual study 
of Kohlenberg and Tsai’s (1991) text, followed by lectures and discussions on the principles of FAP and 
behavior analysis in a small group format. Raters then read the FAPRS manual and were instructed in the 
definition and use of its codes in a small group lecture and discussion format. Raters viewed segments of a 
FAP therapy session previously coded by Rater 1 and discussed the rationale for the codes applied to each 
client or therapist mm. In the next stage of training, raters were taught to code videotape segments taken 
from Dyad A. In each trial the raters observed, coded, and then discussed in small groups each code they 
applied as well as the rationale for their choice. The number of turns coded prior to discussion was 
progressively increased across the training from one (tum-by-tum discussion and instruction) to 
independent coding of an entire therapy session. Across all stages of training, i nf ormal quizzes were given 
to the raters to test their knowledge of the coding manual, their ability to discriminate between codes, and 
their understanding of the client’s functional case conceptualization prior to each training session. No data 
from these training and discussion sessions are included in the reliability analyses presented in this study. 

Rater training cohorts. Raters 2-6 were trained in three “cohorts.” Rater 1 trained Rater 2 in 
the first Cohort. Raters 1 and 2 jointly trained Cohort 2 (Raters 3 and 4). Rater 2 alone trained Cohort 3 
(Raters 5 and 6). The goal of using two different trainers was to ensure that effective use of the FAPRS 
was not dependent upon being trained by the FAPRS’ author and to demonstrate the transfer of FAPRS 
instruction across trainers. This training approach provided a “layering” of training and allowed differences 
in coding s kill s to be examined within and across rater training cohorts. 

Criterion test. At the conclusion of training, each rater was asked to independently code a 
“criterion session” tape. All raters coded the same criterion session tape. The criterion session was used 
to determine each rater’s accuracy in meeting FAPRS training goals by comparing his or her performance 
to Rater 1. If a rater reached the overall (entire session) kappa value set at .60, he or she was allowed to 
begin independently coding tapes included in the data analyses. 

Selection of criterion session. The session used for the criterion test was selected because it 
demonstrated as many of the FAPRS codes as possible within a single therapy session. This session was 
coded in its entirety. Raters were not allowed to code with others present, nor were they allowed to 
discuss their coding of this session. 


Apparatus 


Videotaped Stimulus Materials 

Videotape recordings of FAP treatment sessions were examined for this study. The therapy 
sessions occurred in a room with two comfortable chairs. The video camera was positioned behind the 
therapist and was focused on the client. 

Selection of videotaped sessions to be coded. A sample of sessions from the entire course of 
FAP treatment was selected in order to rate approximately every fourth therapy session. When a 
particular session was of unacceptable quality (e.g., the sound was inaudible or the video was non- 
viewable), the session occurring one week prior to or after the targeted session was chosen at random. 
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This substitution was made if the camera’s view of the client was obscured, or when technical problems 
led to poor audio or video quality. 

Transcripts 

A person not involved in coding the tapes transcribed each of the sessions used in the analyses. 
Transcripts provided a close approximation of a verbatim record of each client and therapist “turn” in the 
treatment session. Transcripts were completed using Mergenthaller and Stinson’s (1992) standards for 
transcription with modifications as described below. 

Embedding of turns. Standards that were modified included “embedding” one and two word 
utterances in the previous speaker’s mm. For example, the client might report, “I became extremely angry 
with my spouse this week,” and the therapist simply responded, “Really?” then the turn would be 
embedded in the previous (client’s) turn and would not be coded. One- and two- word utterances were 
embedded because they often function only to facilitate the previous speaker’s verbal behavior. 

Utterances were embedded only when they were clearly not functional units, such as the answer to a 
question or a verbal reinforcer or punisher of the previous speaker’s behavior. Typical utterances there 
were not functional units and were embedded included turns such as, “Uhh-huh” or “Umm-hmmm” and “I 
see” or “OK.” 


Procedures 


Coding Procedures 

For all reliability analyses, sessions were coded in a randomized order. No identifying i nf ormation 
was included on any transcript that would indicate the order in which sessions progressed, or the point in 
therapy (e.g. early, middle, late) that was depicted for each dyad. The primary advantage to randomizing 
session order is that it prevents rater bias due to raters’ assumptions about how the therapist or client’s 
behavior should change over time. 

The decision to code sessions in a randomized order was conservative, as it would tend to lower 
estimates of the reliability of the FAPRS. Lacking any information as to the point at which the session 
occurred during the course of treatment forced raters to code client and therapist behaviors based only 
upon their apparent function at that moment of the session. When coding sessions out of order, it is 
difficult for raters to determine whether a particular client behavior functioned as an improvement or 
problem. Raters could not invoke rules developed on the basis of previously coded sessions because the 
rater’ s past experience might have included interactions from later sessions in therapy. Although this was 
a methodologically sound decision given the development of the FAPRS to prevent bias in coding, this 
approach made some sessions difficult to code. 

Codes not included in the analysis. The M3 (therapist misses/does not respond to a CRB3) 
code was never used and the ICR (therapist ineffective contingent responding and IN (Generally 
Ineffective Therapist Responding) codes occurred only a few times. Thus, no measure of reliability was 
calculable for these codes. 

Embedding check. Approximately 25 percent of the Dyad A tapes coded for the study were 
selected and checked for appropriateness of embedding by Raters 1 and 2. A list of embedding rules was 
established prior to this check and was used to monitor whether phrases were embedded consistently 
across sessions. A kappa value of K = .91 between raters 1 and 2 demonstrated excellent agreement 
(Fleiss, 1981) between raters 1 and 2 for embedding one and two word facilitative utterances. 
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FA P Adherence Check 

Transcripts from Dyad A were also coded using the Therapist In-vivo Strategies Scale (THISS; 
Parker, Bolling, & Kohlenberg, 1996) to demonstrate that these sessions were indeed FAP session and not 
cognitive therapy. The THISS system is a topographical measure of adherence to FAP strategies. It is 
not a functional measure of FAP competence and was not designed to measure the functional mechanism 
of change discussed above. The adherence check was performed, before testing the reliability of the 
FAPRS, to show that FAP therapist behavior in-session can be distinguished from therapist in session 
behavior in Cognitive Therapy (CT). The adherence check examined therapist use of Cognitive Therapy 
(CT) and/or Functional Analytic Psychotherapy (FAP) strategies within each session to ensure that FAP 
sessions contained behaviors specific to FAP, and more therapist strategies specific to FAP than to CT. 

Reliability Drift Checks 

Reliability decay, or “drift” occurs when a given rater’s reliability to a criterion rater decreases 
over time (Bakeman & Gottman, 1986). This issue is especially important when a series of interactions are 
coded, such as the long series of Dyad A tapes coded by Rater 2. To assess the potential for drift using 
the FAPRS, interobserver agreement “drift checks” between Raters 1 and 2 were performed using coded 
sessions from Dyad A. Insufficient agreement was defined as overall kappa values below K = .5 (“fair” 
agreement; Fleiss, 1981). Failure to maintain criterion at any point would lead to “booster” training sessions 
until the rater obtained sufficient agreement values. Rater 1 checked 25% of the Dyad A tapes coded by 
rater 2 for decay. 

RESULTS 


Adherence Check Using the THISS Scale 

Ratings for FAP-related strategies as measured by the THISS scale were significantly higher (m 
= 1.83 \ sd = .60) than for CT strategies (m = 1.38 ; sd = .20), t(15) = -2.45, p = .027, across the 16 Dyad 
A videotapes used in this study. The mean interobserver reliability for the THISS scale was calculated 
with Shrout and Fleiss’ model (1979) for averaged ratings across a group, yielding consistently high rates 
of reliability between raters (mean ICC = .94, range = .88 to .98). 

Reliability of the FAPRS 

Four types of reliability assessment were conducted on the FAPRS coding system. All reliability 
data for this study are presented in the form of kappa coefficient (Cohen, 1960) values obtained between 
pairs of raters. The “generalized” kappa formula (which yields a single coefficient for three or more 
raters) was not used in this study. The authors decided to present only kappas obtained between pairs of 
raters because the generalized kappa is difficult to interpret. The generalized formula for kappa was 
judged inappropriate for this study because it cannot be directly compared to kappas obtained between 
pairs of raters. Because kappa values cannot be averaged across sessions, dyads, or raters, “overall” 
kappa values were used in this study. The term “overall” kappa indicates that all turns from multiple 
therapy sessions for a dyad were entered into a single, large kappa matrix for each rater. 

Agreement with the criterion rater. Table 3 presents the kappa coefficients obtained between 
each pair of raters and between each rater and the criterion rater (Rater 1) for the criterion session. All 5 
raters reached or exceeded the criterion of k = .60 with the criterion rater. Two of these 5 raters reached 
excellent agreement with the criterion rater. The remaining 3 raters achieved a good level of agreement 
(Fleiss, 1981). 
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Table 3. One Session Reliability to Criterion Tests for Dyad A. 



Rater 1 

Rater 2 

Rater 3 

Rater 4 

Rater 5 

Rater 2 

.83 





Rater 3 

.64 

.56 




Rater 4 

.65 

.61 

.79 



Rater 5 

.74 

.77 

.54 

.56 


Rater 6 

.62 

.62 

.48 

.49 

.71 


Agreement between all other rater pairs. Six of the 10 possible pairs of raters (not including the 
criterion rater) reached or exceeded the criterion kappa of .60 with one another. Two rater pairs obtained 
excellent agreement. Good agreement was obtained by 3 of 10 rater pairs. Fair agreement was obtained 
by the remaining 5 rater pairs. 

Interrater reliability across multiple sessions. Because kappa coefficients cannot be averaged 
across dyads, a separate matrix of interobserver agreement values is presented for Dyads A and B, and 
presented textually for Dyad C below. Table 4 presents the overall kappa coefficients obtained between 
Raters 2, 3, and 4 across three sessions of Dyad A. Table 5 presents the kappa coefficients obtained 
between Raters 1, 3 and 4 across three sessions of Dyad B. 

Table 4. Interrater Agreement Matrix for Multiple Sessions of Dyad A. 

Rater 2 Rater 3 

Rater 3 .60 

Rater 4 .65 .70 


Table 5. Interrater Agreement Matrix for Multiple Sessions of Dyad B. 



Rater 1 

Rater 2 

Rater 3 

Rater 2 

.77 



Rater 3 

.70 

.75 


Rater 4 

.58 

.56 

.61 


The kappa coefficient obtained between Raters 1 and 3 across three sessions of Dyad C was .61. Of the 
10 possible rater pairs in these analyses, 2 achieved excellent agreement, 7 achieved good agreement, and 
1 achieved fair agreement. 

Intrarater reliability. To test the intrarater reliability of the FAPRS coding system, Rater 1 coded 
three sessions of Dyad A at two different points in time. The overall kappa coefficient obtained by Rater 1 
across an 18-month code -recode interval was .91. 

Reliability drift checks. Table 6 presents the kappa values obtained between Raters 1 and 2 for 
each of 7 reliability drift check sessions of Dyad A, and the overall kappa coefficient across the 7 
sessions. None of the sessions ever decreased below a “good” level of agreement. 

Table 6. Kappa Values for 7 Drift-check Sessions of Dyad A. 

Session 1 2 3 4 5 6 7 Overall 

Kappa .75 .87 .68 .64 .64 .60 .61 .72 
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DISCUSSION 


Reliability of the Coding Manual 

Interrater agreement within dyad. The kappa coefficients between each rater and the criterion 
rater presented in Table 3 suggest that raters with varying levels of experience can reach at least a good 
level of agreement with the author of the manual, an expert FAP therapist. The kappas obtained between 
all possible rater pairs indie ate that FAP raters can be expected to agree with each other as well as with a 
criterion rater. This suggests that all raters learned a set of skills for coding FAP, rather than just the 
ability to code reliably with the person who trained them. 

Table 3 indicates that there may have been a rater training cohort effect. Raters 3 and 4 showed 
modally good agreement with Raters 1 and 2 who trained them, and excellent agreement with each other. 
Raters 4 and 5 showed good to excellent agreement with Rater 2 who trained them, and good agreement 
with one another. When compared to one another according to training cohort, however, Raters 3 and 4 
and Raters 5 and 6 uniformly achieved only fair agreement. Interrater agreement is somewhat higher 
within than between cohorts, and some rater pairs agree better than others. It is possible that raters, 
across training cohorts, were making different types of coding decisions and errors. All raters, however, 
remained within an acceptable range of kappa values given the difficulties of functional coding and the 
fact that the FAPRS is still in development. The possible cohort effect will be addressed in future studies 
by using a standardized, computer-based training curriculum for training new raters and for retraining 
experienced raters to a common criterion (Ruckstuhl, Callaghan, & Follette, 2001). 

Interrater agreement across multiple sessions. The across-sessions analyses in Tables 4 and 5 
indicate that good levels of agreement can be consistently attained using the FAPRS across different 
sessions for the same client-therapist dyad. The data indicate that raters can sustain these good to 
excellent levels of agreement across several sessions. These data also show that raters can achieve high 
levels of agreement for different client-therapists dyads; that is, rater agreement levels generalize across 
dyads when coding different clients and different therapists. 

Assessment of the reliability of the FAPRS for measuring client and therapist behaviors across 
multiple dyads is challenging and important. The FAPRS requires raters to discriminate and correctly 
categorize both client and therapist behaviors based upon the approximate function that each dyad 
member’s behavior has on the other person. This can be a difficult task, as each dyad has a 
topographically different means of creating a common function within the therapeutic relationship. 
Topography can also change within a particular dyad across time. While two FAP therapists might identify 
“trusting the therapist” or “feeling close to the therapist” as CRB2s in the functional case 
conceptualization for two different clients, the way that each client’s CRB2 looks is typically quite 
different, making coding across clients difficult. 

The fact that kappa values from Dyads B (Table 5) and C (kappa = .61) matched or exceeded 
those of Dyad A (Table 4) suggests that using sessions from Dyad A to train the raters did not confound 
the results for the reliability analyses. Future studies of the FAPRS will train raters with therapy examples 
from multiple dyads, and to the greatest extent possible, use training materials taken from dyads not 
included in the data set of the study for which raters are being trained. Given the status of FAP as a 
relatively new psychotherapy practiced by a limited number of clinicians, it was necessary to train raters 
using certain dyads. It is encouraging that interobserver agreement rates in this study did not appear to be 
bound by rater training that used one of the client-therapist dyads included the primary data analysis. 
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The data provided for the intrarater reliability check suggest good test-retest reliability for the 
FAPRS. These data should be interpreted cautiously. The intrarater kappa coefficient represents the 
coding of one rater (the criterion rater) examining only one client-therapist pair. These data are presented 
here because they suggest that raters may be able to use the FAPRS consistently across time. This is a 
promising finding given the length (18 months) of the code -recode interval used and the challenges 
imposed on the rater by the complexity and subtlety of the behaviors coded. 

The data in Table 6 for the drift check analysis indicate that two raters can use the FAPRS, 
across multiple sessions with good overall reliability. Overall, Rater 2 was able to sustain sufficient 
agreement with the criterion rater across the sessions used in the data analyses. The session-by-session 
kappa values indicate that reliability was subject to slight decay. From these tables, we conclude that 
ongoing training in conjunction with reliability checks may be necessary, and that future studies of the 
FAPRS should continue to include both session-by-session and overall kappas when examining rater drift. 

Testing the Hypothesized Mechanism of Change 

The reliability data presented above suggest that the FAPRS is reliable and that an empirical 
examination of the therapeutic relationship and FAP’s proposed mechanism of change is now possible. In 
this section, a brief example of using the data from the FAPRS to test the hypothesized mechanism of 
clinical change will be given. 

Brief description of lag-sequential analysis. Sequential analysis uniquely captures what occurs 
in complex interactions (e.g., Bakeman & Gottman, 1986; Gottman & Bakeman, 1979; Gottman & Roy, 
1990) and is especially well suited to examining what occurs when two people interact in the context of 
therapy. Lag- sequential analysis compares the conditional probability of a targeted event, given the 
occurrence of some other event, to the simple probability of that targeted event. To test whether the 
mechanism of FAP occurs as hypothesized, researchers can use data from sessions coded with the 
FAPRS to determine whether the therapist responds contingently to in-session client behaviors. 

Answering this question helps determine whether the therapist is engaging in the behavior necessary to 
bring about change. For example, given a CRB2 by the client, we can determine whether the therapist 
provides a contingent response following that CRB2 (TCRB2) more frequently than he or she emits a 
TCRB2 independently of the client engaging in improved behavior. A z-score and its associated 
probability are calculated for these comparisons to determine whether these two are statistically 
significantly different. Interpretations of a significant z-score are based on the assumption of the null 
hypothesis (that there is no sequential relationship in the data). 

Analyses. The data used for these sequential analyses consist of 1,788 turns from 31 sessions of 
Dyad A coded by Rater 1. Transitional probabilities used in the sequential analyses and corresponding z- 
scores were calculated using Allison and Liker’s (1982) equations for computing lag sequential analyses. 
The use of z-scores controls for the base rate, or unconditional probability, of the target code. Z-scores 
demonstrate the degree to which the target code is contingent on the occurrence of the given, or specified, 
code. Z-scores above 1.96 are considered significant at the alpha = .05 level and indicate that the target 
code occurs more often relative to the given event than would be expected, given the unconditional 
probability of the target event overall. Negative z-score values indicate that the target code occurred, 
statistically, at a level less likely than expected. 

Lag- sequential analysis of coded client and therapist behaviors tested the hypothesis that the 
therapist actually engages in contingent responding; that is, the therapist engages in the behavior necessary 
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to effect client change. Data for this analysis are presented in Table 7. Table 7 shows the z-scores for 
the conditional probabilities for lag 1 of the therapist’s responses to client problem behaviors in -session 
(CRB1), client improvements in-session (CRB2), and client identification of relevant controlling variables 
(CRB3) across all 31 sessions. Transitional probabilities are presented for the therapist missing CRB Is 
and CRB2s (Ml and M2). The table also indicates the therapist’s positive session progression behavior 
code (TPR) in response to the three antecedent CRBs. 

Table 7. Results for Sequential Analysis for Therapist Behavior. 

*overall z-score for 5165 events, significant atp < .05. 


Therapist Behavior 


Antecedent 

TCRB1 

TCRB 2 

TCRB 3 

Ml 

M2 

TPR 

CRB1 

53.81* 

-2.87* 

4.70* 

15.98* 

-.65 

-9.16* 

CRB2 

-2.91* 

57.88* 

-.38 

-.66 

15.84 

-5.21* 

CRB3 

-1.37* 

-.79 

14.66* 

-.31 

-.18 

-.78* 


These data demonstrate that at lag 1 , the therapist was more likely to respond to a CRB 1 than 
engage in other behavior. While the therapist also missed some CRB Isa test performed on the difference 
between the z-scores (calculated using Z=(Z\- Z9/V2; Rosenthal & Rosnow, 1991) for these two codes 
revealed that TCRB1 occurred more often than Ml (Z= 26.75, p < .05). The therapist was also more 
likely to respond to a CRB2 than engage in any other behavior, and a test between z-scores for TCRB2 
and M2 was statistically significant (Z = 29.73, p < .05). The therapist was again more likely to respond to 
a CRB 3 than engage in any other behavior when a CRB 3 occurred. The negative z-score value for 
TCRB2 given a CRB1 indicates the therapist was unlikely to respond to the CRB1 behavior as though it 
were a client improvement. In the same manner, the therapist was statistically less likely to respond to 
CRB2s with a TCRB 1 , indicating the therapist did not respond to CRB2s as if they were client problem 
behaviors. 

The lag sequential and z-score comparison results indicate that the therapist responded 
contingently and effectively to CRBs as required by the hypothesized model. The therapist was more 
likely to respond to the impact of the client’s behavior appropriately (e.g., emit a TCRB1 when the client 
engages in a CBR1) than engage in either an incongruous or ineffective contingent response (e.g., emit a 
TCBR2 in response to a CRB2) or engage in general therapeutic progression responding (non FAP- 
specific behavior). This finding provides initial evidence for the presence of the mediating mechanism of 
FAP and the use of sequential analysis as an investigative method for researching mechanisms of change. 

Relating mechanism of change to outcome. To relate this responding to changes in client 
behavior, future studies of FAP’ s mechanism of change can compute the simple probabilities with which 
clients engage in CRB Is, 2s and 3s can be examined across a data set. Data can be divided into thirds of 
treatment, identified as Early, Middle, and Late thirds, and the combined simple probabilities for CRB Is, 

2s, and 3s, can be calculated for each third of the treatment by summing the total events for each code 
and dividing by the number of client events for that third of treatment. Chi-square analyses can then be 
used to demonstrate the changes in proportions (simple probabilities) of CRBs during each third to test the 
hypothesis that problematic behaviors (CRB Is) will decrease and improved behaviors (CRB 2s and 
CRB3s) will increase over the course of treatment. 

Implications for Future Research 

In an ongoing effort to improve the training of FAP therapists, the coding system described in this 
study is being used to train therapists in the theory and practice of FAP. The FAPRS is being used to 
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teach therapists to accurately identify their own, as well as clients’ behaviors in the sessions they conduct. 
This teaching method provides a technology to empirically document the change in therapists’ behaviors 
over the course of their training and their corresponding effectiveness in bringing about a change in their 
clients’ behavior. The reliability analyses presented in this study are only a first step toward the refinement 
of observational measures for FAP and eventual studies of the validity of FAP’s proposed mechanism of 
change. Reliability studies are a necessary stage, however, and these initial data are encouraging. 

Future studies of the FAPRS must address several weaknesses, including the potential for training 
cohort effects and possible confounds when training and testing of the FAPRS uses the same dyads. It is 
likely that a diversity of behavioral exemplars will be required for raters to become well trained in a 
functional coding system such as the FAPRS. Coders must learn to more quickly and accurately extract 
information about behavioral function regardless of topography. The development of standardized, 
computer driven training systems and expansion in the number and diversity of FAP dyads available for 
study are key steps toward addressing the weaknesses of this study that are being pursued currently. 

Conclusion 

This analysis of the Functional Analytic Psychotherapy Rating Scale provides evidence that the 
functional variables hypothesized to be important to FAP’s mechanism of change can be reliably observed 
and documented. The FAPRS coding system will eventually allow clinicians and researchers to test 
hypotheses about the therapeutic relationship and its role in effecting clinical change in a manner similar to 
the example provided above. Other psychotherapies may be able to use the methodology described in this 
study to specify and measure important elements of the therapeutic relationship and their influence on 
client behavior change. Although this research is in its early phases, it lays the groundwork to explain 
variance that remains unaccounted for in traditional psychotherapy outcome studies, namely, “What occurs 
in-session that helps clients improve?” 
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